DOCUHBHT BESOHE 



Bfi 112 812 

AtlTHOB * 
TITLE 

INSTITUTION 

SPONS AGENCY 

PDB DATE 
NOtE 

• 

EDBS PBICE 
DESCEIPTOBS 



HE 006 885 

» 

0 

Gjerde, Craig !• ^ ' ^ 

A Data-Based Approach to Setting Performance 
Standards. ' ^ 

Connecticut Oniv. Health Center, Farnington. Dept. of 
Research in Health Education. 

National Fund for Medical" Education, Cleveland, 
Ohio. 
Apr 75 . 
18p. 

MF-$0.76 HC-$1.58 Plus Postage 

♦Behavioral Objective's; Core c^urriculum; Curriculua 
Developnent; Curriculum planning; JKicational 
Objectives; Faculty; *Higher EducaTlon; *Medical 
Education; Medical Students; *Performance Criteria; 
♦Teacher Participation 



ABSTRACT 
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for the evaluation of stl^dent 
benefit of this systemato-c de 
important to the institution 



o developing a core cafriculum with the 
tracking System, the /Committee on 
on at the University^ of Connecticut 
mitt ee that set out: to fin4 a systematic 
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b^xs in writingVand reviewing 
ves (affective/ cognitive ^ and 

standaxds developed by the 24 teaching 
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' ' INllBlC/fiUCTlON: A THREE-YEAR PROGRAM? 

A few years ago there, was an Idea that medical schools could produce 
students with a '*core** knowledge of medicine ii^^ree years rather than \^ 
In four years. This plan, the argument went, J^uld enable students to 
proceed earlier into a specialty track and thus enter practice sooner. / 
The federaJL government was encouraging such three<-year programs and of- / 
fering special funds^o schools which participated. ^Some medical schools 
responded to the financial inducements to design three-year programs; 
other schools ^^^^ cautiously debated whether "^^uch experimeats-might 
lead to poorly jprepar^ doctors and wondered whether their programs 
could be tightened up to produce ec^ually competent doctors in a shorter 
time. ♦ ^ 

/The Comnlttee on Undergraduate Medical Education at the University 

J ' ^ 
of Connecticut Health Center decided that the b^st approach was to appoint 

a committee called'^the Core and Tracking Cotpdaittee to explore with faculty 

\ ■ •■• -\ 

from out^24 teaching committees the core competencies required in each 
pf them and to make recommendations concerning mechanisms^ for increasing 
elective and tracking opportunities. 

'f 

The Core and Tracking Committee set out to determine if , there was 
a systematic way of looking at out curriculum. Our long-range goal was 
to collect data, using a judgmental approach, whiph would provide a 
rational basis for developing a core medical curriculum with the pos- 
sil^ility of instituting a tracking system. As you might suspect, this 
study lead us into institutional politics and forced us to become chapge 
agents. . ^ / 

I would like to describe the medical school environpient, our school's 
needs for a more explicit curriculum, and finally the methodology we used 
to develop "^performax^ce standards. 

' V ■ / ■ ■ ■ ■ ■ . 

' 3 ^ ■ 



i ' - PLANNING 

l... , . ■ ■ • ; . . 

1 Ijlie eArly Stages of the work on the project involved getting to know 

■ . " I) ' ■ , • • 

the^iiedical environment and planning the best ways to proceed in getting 
t:hc work done. A ^jor concern was that we insert ourselves carefully 
into the environment, since we had to(wih the cooperation of the faculty 
and could not force anyone to cooperate with us. Thus our first step 
was to conduct a round of interviews with some of the teaching committee . 
chalrmc|n, academic departmental chairmen, and members of our Core arid ^ 
Tracking Committee. We felt that it was important for them to hear our 
Ideas about an explicit curriculum and for us to sefe the educatidnal ^ 
system of the Health Center through their eyes. 

Durlns^ these interviews we gained an undeirstanding of the teaching 
organization and heard faculty opinions about what was rijiit and wrong . 
about our program and got their opinions about a "core" curriculum. 
There was much contentment with the status quo, and niany committees 
felt that theJ^r courses* already consisted of a minimal core. We found 
that in our educational program <as .in many medical education programs 
which are organized around organ sy^frtems, rather than the traditional 
academic departments) the teaching program floats with little organizational 
strupture. Tqac^iing committee chairmen are selected from the general 
' faculty with no concern for departmental base; similarly persons from 
various departiriftnts are asked to teacR on these committees. This ^ 
reslilts in the <S\lairman not always knowing the whole content 
which he is the 0Uairman. It also means that the fajyrirfy who teach on 




the committee may not know each other and may^o.t even see each othe^ 
to plan the course ^- they may jus^x^-fill in ^lecture slots in the syllabus. 

Heading a subject coirajj^ttee was a burden not always relished by 
tL subjjBct conmittee €>dirman* A faculty person was rewarded institutionally 





and nationally for his research; ed^ational yn^tters demanded his time 
(in planning the course, developing schedule^, writing tests, finding 
lectures and preceptoirs for labs, determining curriculum, ordering mouies 
and library materiald, attending meetingS| tutoring students with problems 
etc,) and provided few rewards. Being a chairman demanded perhaps a 
raonttfof planning time, pl^us the hours which the chairman was expected to 
spend in lecturing and in attending other lectures • 

With many educational responsibilities, the chairman was often 
forced to leave the course content pretty much the way it had been in 
previous years; curriculum revision normally received little attention in 
committee meetingf. The curriculum was ,in5)erfectly defined by a book 
called the syllabu$v This book listed. th0 topics for .each of the class 
/sessions and 6"f ten contained outlines of the subject matter, supplemental 
readings, and self-study materials^ Often the chairman was not an 
expert in all the subject ^mdtter taught in his organ-centered committee, 
so he frequently relied the individuals who taught on the committlae ^ 
to determine theit* mrci curriculum content. -At worst, curriculum segment s^ 
were determined by that one individual with tio systematic provisionals 

ew by other faculty. At best, th^ chairman and two or three others " 
planned the appropriate^glrfbal^^ for the course atid tried to focus 

lecturers to tho^er^'opics. yath clinical areas, students learned from 
preceptor^^'^wl^se concept ^^ppropriate curriculum content was usually 
even explicit and pibre varied. 

0 

'i 

The subject chc^irman had no power either to reward or to punish; he 

/ ' ' .» 

had to operate op the basis of professional responsil^ility, personal 
respect, and |tiendship. In af^ift^lel s:^uation, the project director 

|he w)|rked; he had to make his\;ase, 
he chairmen (^nd through them 




powfir over the pieople with 
he could and try to conv 




their t^^ching colleagues) that this effprt of determining performance 
flttadards was worthwhile. 

' NEED FOR AN EXPI^lCIT CURRICULUM 

As we cpndutted our interviews we built up a rationale for writing 
objectives. It appeared that in addition to serving the long-range needs 
of the Core and Tracking Committee, determine course objectives could 
be helpful to many constituencies at the Health Center* 

S tudents 

]^ocus on material : Students ap® often confronted with huge syllabi 
full of information whict they are supposed to "knows" Objectives can 
help them to i^ocus on the important topics of study and help them to see 
some overall organization of the material. 

Teaching Faculty ^ 

Curriculum definition : Specifying objectives is a way of defining 
a curriculum in an explicit way, tather' than in the implicit way pre- 
ferred by many faculty. The^ explicrit curriculum — as defined by a list of 
objectives can be the basis for rational deliberation in which the 
jdiscusslon can be focused on specific points • In the past curriculum 
guidelines often consisted of lists of the course content with^p^^ndication 
of what the student should be able to do with that content. 

Student evaluation : Similarly the faculty can ask "Wh^ is the 
best way to evaluate whether the student is attaining objective?" 
The list of objectives can serve as a basis for evaluating the student's, ' 
skills and knowledge, and for making recomlnendations to the student for 
additional work. Attitudinal objectives can provide a basis for narrative 



0 



/ 

^ev^Xu^tlve conmeiitQ which we use to supplejient on pass/fall grades. 

\ Course integrity ; Perhaps ^the bes^ reason for having objectives 
ifl^^o faculty can ^pec^ffy what is important for the students to know ^ 
And ^i^erform, validate that decision by a democratic polling, and then 
CQope^ate in trying to attain those objectives. Unless faculty are ^ 
in agijeeroent in what they are trying to do, and unless those things which 
they.^re trying to do are spelled out and communicable, it is more 
likely. that faculty will take a shotgun approach to the presentation of 
Bacterial and will not successfully integrate it into a meaningful, 
y^ourse. Only after objectives of a program are delineated is It possible 
to determine priorities among those objective^ through systematic ratings 
by the teachers and by other professionals. 

Having identified the needs ^ for curriculum definition we had to 
determine a format for stating the objectives, decide on the language 
to be used in stating the objectives, and plan the later steps in the 
process. 

fl 

METHODOLOGY 

* 

Format for the Objectives 

After some review of the materials on writing objectives by Mager, 
Gronlund, and others we, developed a format ij^r objectives: . ^ 0 

1. All objectives(woutd state the acts which ^STUDENTS would be 
1 required to accSmplish^ — The implicit stem of each objective^ ^ 

would be "the student wilj) be able to .\ . v 

2. Objectives would be time referenced to th^end of the course. - / 
Thus there was an implicit "at the^end of this course ..." 
associated with each statement. > , i 

3. The body of the objective would he composed of two essential 
parts - a behavioral verb and a bit of content. The verb 
defined the way in which a student might b ^^appropriately required 
to demonstrate his achievement of the content which was speckled 
in the objective. 




4« fho behavioral verba could not be "knowj^' "understand," of 

my One of several listed verbs vliich have multiple Inter^ - 
prctations. The verb had to be more of a behavioral -sort, 
•uch as those found on a li.st of suggested verbs which we; 
providedM:o writers, (Handout) 

/ 

Objecttves were defined as descriptive statements which lay soii>ewhere 
in specificity between the goal statement of "knowing biochemistry" and 
the specific statement of a fact to be recalled. Faculty were encouraged 
to cdnsider three types of objectives: affective, cognitive, and psycho- 
motor. Affective objectives were especially encouraged,v 
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Scales ^for Ranking Objectives 

k(te7> some deliberation of the dimensions on which objectives could 
be rajzed, we decided that objectives should be rated on their CLARITY 
Qi.e,, ability to clearly communicate to the reader the behavior which 
the student was expected to display in achieving the objectiv0 ^d on 
their CENTRALITY (i.e., the essentiality for ev^ry student in the program 
to achieve th>f objective) . (Handout) , 

Cl arity ; JEach judgmenx of claritYr includ^ a reconjmendation to the 



Conmittiee on wh|(t to do/^o iinprove the/8ijjective. The choices were: 
JUDGMENT / ' ) RECOMMENDATION 




Very iC^rfear V \ Lea>?ii-it alone 

Quit^lear ^ , Cbald tighten it up 

Varied Interpretations Possible Ne^ds Wbrk 
Somewhat Unclecir Revise' 
Very Unclear / / Rewrite 



Centrality : The ot;Ker/scale^on which ratings were made was the scale 
on which reviewers could rate the-^entrality or essentiality^ of each r 
objective*. The question was "how essential is it for every student in 
our prograipfc^ achieve this objective?" The choices were: ^ 



Essential for all students 

Desirable but not essdnitial 

Useful but should not: be required 

Unessential 

I am unable to judge 

^ ^ 6 - 



This first rating waa done only by members of the committee whose 
objectives were being written and was done as a way of helping them tb 
identify the objectives with which they did not agree on the appropriate 
content depth or. student behavior. As faculty looked .£tt the objectives - 
thcy^^were invited to modify objectives, delete objectives, and add 
objectives. 

Sinee the writing of the objectives was frequently done by small 
groups, this rating ]^rocess allowed all committee members to participate 
and infprmed them of the curriculum deliberations going on within their 
committees. (Handout) ^ * 

Working with Faculty / ^ I 

We decided to be fltoible and'all^ow serious deviations from the 

•) . ■ " - . ' 

initial pflan. As it turned ^ut, it was. easier to acconqf)lish the task 
of getting Objectives written by ignoring much of the initial plan: 
the chairman .of the cofimitte^ and two^^her persotis seenjed about the 
right number to^wotk with; explaining objective writing and the cognitive, 
affective, and psychornotbr domains took. only 10-20 minutes. Most groups 
did not need to devote any timfe exclusively to determining course content; 
course content appeared to be^well .fixed in their minds. Deciding the 
appropriate expectations- for ^$tudents was more difficult, yet the actual 
task of writing the objectives did not seem iis difficult as most people 
had expected. Four to 'ten hours were. required for an overage six-week 

f r 

committee, dnd three hours at one stre*tch seemed' tb be the maximum 
tolerable time for, this kind of mind searching. ^ 

W^ tried to avoid makltUg the working sessions a^ tfh^e for debating' 
the prqs and cons about writing obiectives; we merely ^chaJ^nged them to 
tell us what it was that they expected a student to be abld to do by the 



end of thcif course-- or conversely, vhat tKey would be 'unhappy^ ab'out 

if students could not do it by the end of the course, ^he project • ^ 

director hcted as secretary /catalyst/provocateur and wrote down theil i 

/ • ■ i 

statements as they thought them oui, / 

In some cases the faculty ^ferred to/a^ copy of their syllabus 
or a class sched/ile to refx^j( their memories. Often the emergence of ^ 
an objective \p)uld l^e prefaced by deep thought or discussion about Just 
what the student was really supposed to learn. Frequently the persM[ 
would struggle for a miijute in his mindy/ith an objective, then choose 
a verb from the verb list to describe the way in which, students could 
be asked' to demonstrate their knowledge. Most faculty soon recognized 
the groupings of cognitive verbs into hierarchial categories, 

Any ob;Je(gtive considered important by the faculty person was accepted, 
although the clarity of the statement would be questioned if it were- 
unclear or confusing to the director, " ^ 

i ■ " I ' : • • 

^ STATUS ! 

■ i 

In two years of part-time effort almost all of our 24 committees v 
^ve prepared objectives. Five or six committ/ees have published these 
objectives in^eir' syllabi. Some cdramitteed are using their objectives 
to develo^erformance standards for student evaluation. This phase of 
the worjc will continue during the next two years, 

;n summary, the day-to-day benefits from 'this systematic clefinition 
curriculum may be far more important to the institution than their 
original purpose of considering a core program with possible early ^ tracl^ing. 
The interaction of staff in rethinking courses and writing objectives has 
resulted in statements of the educational objectives of committees, but 
in a more iraiiortant senlse it has caused a fieven-year-old institution to 



rethink Its educatioipi^ goals, Its committment to education, and 
perhaps to reorganij^e its educational system and its reward system. 
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HEALTH CENTER 



14 August, rJ,974 



bear Bios tatis tics Partici]^ant, 

■ ■ ' ^' ' • ■ ' ■ • I- 

The Biostatlstlcs slibject coxamlttee is cooperating with the Core 

V . . , , ■ "■ ' . ' . . ^ ; ••' 

Viand Tracking Subco^ittee of CUME in specifj^ing and obtainin^g ratings oh 

' * its educational objectives, We^ never did get arouhd to rating a final 

list of objectives. Would you please i^ntify any miJsrsing objectives^ 

iQiDdify objectives which might be poorly focused^ and filially rate each 

of the object ive9 in t^^rms of its clarity and in terms of i^s c entral lt^ 

<^r importance. " V 

■■ * • ' 

/ — These are the scales on which you are asked to rat^ t5he\ objectives: 



Claflty of Statement 

\How fciearly does this objective 
Communicate the behavior which 
the student is expected to dis- 
^llty in achieving the objective? 



Vety 
Clear 


Quite 
Clear 


Varied 
Interpre- 
tations 
Possible 


Some- 
what 
Unclear 


Very 

Unclear. 


Leave 

it 
Alone 


Could 
Tighten 
it up 


Needs\ 
Woric 


Revise 
It 


Rewrite 

it ' 



Centrality in l/CHC Curriculxm^ 

How essential is it foy every 
student in ou^ program to 
achievci this objective? 



ESSENTIAL 
Jor all 
isttidents 


DESIRABLE 
but not 
essential 


USEFUL 
but should 
not be 
reaulr^d 


UNESSEN- 
TIAL 


I am 
unable 
to 

judge _ 





After the. suggestions and ratings are^ considered, the objectives will 
. ' ■ - ' ^' 

be share'd with other subject committees. Please return your ratings in the 

envelope provided by mid-Septembers . 

Craig GJetde 



: 1/ • ■ ' ■ . ' . ■ . ■ 

Adapted from Bruce Spivey, "A Technique to Determine Curriculum Content," 
Journal of Medical Education . 46 ; 2^5-WWZl . 
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tnstruGtions: AftVr you read each objective, rate its 
:larity and ♦ its * c6ntrality . Feel . £ree\ to revise any 
3b j active ati\to"^dggest additional obi^ective^. 



RATINGS OF OBJECTIVES 



Kame (optional) £_ 
Date: 



UueacIooAl ObJaccivM 



UNIT I 
1 



pa fine tht chraa 

1. Kean 

2. Ke4iafi 

3. Koda 



Dtfint 
1. 
2. 




laauraa of ctntral tandancy. 



Mflauret oC ditpettion. 
Deviacioff^ 



4. 
S 



Given a ate of nuribcrp calculate both by hand ond by ualng • 
ealculetoc etch of tho following : 

1. Kcan , ^ ' - , 

2« Variance - ^ « 

9« Standard Devlatiou . '\ 

itete the aaaential paraaetors needed to define e nerval curve. 

Civcn the appropriate infonnetlon, trensfone e rew score into its 
•tendard (Z) score and vice verse. 

Tranelete values of Z ^o erees of the norwil curve end 
Vice verse. \ . 

Describe end centres t the three ecelee of BeesurcBeot: 
1«— Moainal (Discrete) 

2« Ordinal (Ranked) ^ ^ ^ 

3« Intervel (Continuous) * ' . ' 

Distinguish between clinicel end epidcmiologicel 
•tudlee*. 



r 

2. 



10 



UrfXT 2 
1 
2 

3 



S 



Cleseify cpidemiolocy reeeerch into prospective dr 
retrospective designs. 

.Civcn tho following points from the Schor erCiclc: 1.,. 5, 6» 
t. 9a 10» 11, end 12, explain them in your own worde. Or 
given e short example, idsntify the major faults according 
to Schor. 

S 



Determine whether. tw6_e vents are depejndent or independent. 

Apply the addition and product roles for calculating 
probabilities. 

Detormine whether a given variable is distributed according 
to tho binomial distribution. 

Calculate probabilities using the binomial formula. 
Example: What is^the probability that a random selection 
of 10 people will Vield 6 with blue eyes if the probability 
of blue eyes is 0.4 in the general populatioa. 

Calculate the mean and variance of a binomial distribution. 

Use the normal distribution, to approximate binomial 
probabililtles. • ^ 
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aa chla aVjaeCtm 






iteaca tha b«havlar whlck 














ymxM 












SaM- 




v«>nf 


Quit! 


tadana 


what 




CUaf 


£l£i£ 




Unclaar 


Uncliar 




Coul4 


KuaJs 


r<2«lia 


UwtlC* 


AX 


T(|btaa 


WarV 


♦t 


Ic 




It 






















^ "i 






9 


8 


7 


6 


5 


9 


8 
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5 
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8 


7 


6 
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8 ■ 


7 
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8 


7 


6 
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8 
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5, 
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8 


7 


6 
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8 


.7 


6 


5 


9 

✓ 




7 


6 




9 


8° 


7 


6 


5 


9 


8 


7 


6 


5 
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8 


7 


6 


5 
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8 


7 


6 


5 


9 

t - 


8 


7 


6 


5 


9 


8 


7 


6 


5 



*»v aaamtl*^ i« Ic far tvaff 
■tiMlcoC la aiir vcagru ca 
sckiava UU abjactival 



esszvrtAL 

far all 
atiilantt 





USETVIr 


umtD-i 


but MC 


kuC ahauM 




aaaaattal 


MC ba 











s 



3 
3 



U 3 
4 3 



3 
3 



3 
3 



3 
3 

3 



2 
2 

2 ' 

2 
2 



■ \ 

2 



2 
2 



2 
2 



( 



// 



' B-2 



V 



DO NOT WRITE ON TESTS 
USE PAPER PROVIDED 



4. 



5. 



■4 



Which of vthe followinc^ attributes areLd^stributed according 
"* to the binomial disttibution. Explain your answer.' 



A. / se!x , 

B. r race 

b. ^ ' hair color 




E/T hand used to 

brush your teeth 
weight 



If the 'five year survival rate for a particular disease 
is %60/what is the probability thai: exactly 2 of 3 patients 
with the disease wil^i be alive at the end pf five years. 



in the general population, 12% of. all people afe left 
handed. In a randoin^ple of lOQ. people , wha.t is th^ 
probability of bbtad^ning 80 or less right-handed people? 



using the binomial formula rl(S-r)l 'P^'q"^'' . ^^^^ Z"^^' 
probabi^ty that "at least .2 of the 4 patients a physician 
is tJfeating will survive if the fatality rate for the 
<3irsease is 0.25? 



r 



Of 10 patients with a particular disease, what is 
expected mean number of survivors if the survival 
is 0.35? Also calculate the variance and gtandard 
deviation for this distribution. 



the 
rate 



NOTE? On problems involving calculations ,^t is only necessary 
set up the arithmetic. 



I 
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MEDICAL SELECTIVE 



EDUCATIONAL GOALS AND OBJECTIVES 



q)ALS '• 

1. UnderstandN^th^^inciples^ practice, and scope of the specialty: of 
Internal Med}.cine. 

2. Undfer stand the correct approach to and the medical management of a sick 
adult. 

OBJECTIVES 

!• In^p^detitly define the 1 critical questions about the patient's episode 
of illness ;^|stiggest means by which the questions qould be answered; try to obtain 
the information to answer the cticical questions, ' 

i \ ■ ► - •■ . 

?• Eilcit histories that are complete to t^he point where a more experienced 
physician cannot consistently obtain relevant additional data from the patient at 
roughly the same point of time. 1 

3. Obtain additional information aboiit a patient from other sources when it 
i is necessary to do so. 

4. Perform physical examinatit>ns that are generally complete and accurate 
within a reasonable period of time. 



5. Perform certain labqratory 



, ^ tests (e.g., examination of a peripheral 

blood'smear, determination of the hjematocrit, and a complete, urinalysis) routinely 
on all patients worked up by students, and other tests (e.g., gram stain of body 
fluids) on selected patients when ladicated. . . 

^ ^ ■ \ I 

6. Prepare a complete, accurf'te, afid appropriate problem list expressed Un, 
words that reflect the degree 'Of definition of the problem. ^ 

?♦ Interpret or assess each tttem in the problem list, and provide an appropriat 
differential diagnosis and formulitipn wherever, they are indicated. 

8. Develop a plan of management that contains appropriate diagnostic, thera- 
peutic, and patient education elements; carry out this plan responsibly when it 
has been approved; explain the michanism of action of the drugs used. 

9. Justify your diagnostil and therapeutic plan by assessing the benefits 
and risks of each element of the plan. 



10. Communicate tactf ulljilwith the patient using language that he Vs most 
likely to understand, and demoijstrate a degree of sens^d:t?i^vity toward the concerns 
of^the patient and his family. 
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ia. Tactfully explain to the patient and/or his family. the nature of the . 
l llnesaj its e valuation, treatment, prognosis, and any alterations in life styl# 
^^ch iV necessitates- / \ 

12.\ FolLdw thfe patient closely throi^ghout hdiiS hospitalization in order -to 
understand, insofar (as it is possible, what is going on with tha£ patient^^. / 
It* any poirit in time; revise formulations wlj^enever new Wid ence ^war r en tj? doing t 

r ■ ;, \ ^ ^ „ . , ' 

' 13. Write appropriate problem-oriented progress' notes in the #hdrt. 

\ 14. interpret all available diagnostix^Jjvfd^rmation about a patient, e.g., 
xirrays, biopsy material, lab tests, prb^jedilfe results such as sigmoidoscopy 
'f^nd^gs. n ^ . ^, 

^ 15. Write appropriate orders using tlie cor^f^^ format. > 

r6^ Perform certain procedures, e.g., venipuncture, arterial puncture," 
tho|:^coceAtesis, lunibar puncture, EKG. * / ' 

17, Write a ^schar^e. summary. , 

18. Interact/ appropAately yith peers and other members of the health 
care team. / r 



/ 



< / 



% 
ft. 
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